Displaying sound indications on a wearable computing system

ABSTRACT

Example methods and systems for displaying one or more indications that indicate (i) the direction of a source of sound and (ii) the intensity level of the sound are disclosed. A method may involve receiving audio data corresponding to sound detected by a wearable computing system. Further, the method may involve analyzing the audio data to determine both (i) a direction from the wearable computing system of a source of the sound and (ii) an intensity level of the sound. Still further, the method may involve causing the wearable computing system to display one or more indications that indicate (i) the direction of the source of the sound and (ii) the intensity level of the sound.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/295,953, filed Nov. 14, 2011, entitled “Displaying Sound Indicationson a Wearable Computing System,” the contents of which are fullyincorporated herein by reference.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Computing devices such as personal computers, laptop computers, tabletcomputers, cellular phones, and countless types of Internet-capabledevices are increasingly prevalent in numerous aspects of modern life.As computers become more advanced, augmented-reality devices, whichblend computer-generated information with the user's perception of thephysical world, are expected to become more prevalent.

SUMMARY

In one aspect, an example method involves: (i) receiving audio datacorresponding to sound detected by a wearable computing system; (ii)analyzing the audio data to determine both (a) a direction from thewearable computing system of a source of the sound and (b) an intensitylevel of the sound; and (iii) causing the wearable computing system todisplay one or more indications that indicate (a) the direction of thesource of the sound and (b) the intensity level of the sound, whereinthe wearable computing system conditions causing the wearable computingsystem to display the one or more indications on the intensity levelbeing above a given threshold level.

In another aspect, a non-transitory computer readable medium havinginstructions stored thereon that, in response to execution by aprocessor, cause the processor to perform operations is disclosed.According to an example embodiment, the instructions include: (i)instructions for receiving audio data corresponding to sound detected bya wearable computing system; (ii) instructions for analyzing the audiodata to determine both (a) a direction from the wearable computingsystem of a source of the sound and (b) an intensity level of the sound;(iii) instructions for causing the wearable computing system to displayone or more indications that indicate (a) the direction of the source ofthe sound and (b) the intensity level of the sound; and (iv)instructions for conditioning causing the wearable computing system todisplay the one or more indications on the intensity level being above agiven threshold level.

In yet another aspect, a wearable computing system is disclosed. Anexample wearable computing system includes: (i) a head-mounted display,wherein the head-mounted display is configured to displaycomputer-generated information and allow visual perception of areal-world environment; (ii) a controller, wherein the controller isconfigured to receive audio data corresponding to sound detected by awearable computing system and to analyze the audio data to determineboth (a) a direction from the wearable computing system of a source ofthe sound and (b) an intensity level of the sound; and (iii) a displaysystem, wherein the display system is configured to display one or moreindications that indicate (a) the direction of the source of the soundand (b) the intensity level of the sound, wherein the controller isfurther configured to condition causing the wearable computing system todisplay the one or more indications on the intensity level being above agiven threshold level.

These as well as other aspects, advantages, and alternatives, willbecome apparent to those of ordinary skill in the art by reading thefollowing detailed description, with reference where appropriate to theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart illustrating a method according to an exampleembodiment.

FIG. 2 is an illustration of an example microphone array, according toan example embodiment.

FIG. 3 a is an illustration of an example display, according to anexample embodiment.

FIG. 3 b is an illustration of example arrow sizes that may be includedin the display of FIG. 3 a, according to an example embodiment.

FIG. 4 a is an illustration of another example display, according to anexample embodiment.

FIG. 4 b is an illustration of an example color-coded scale, accordingto an example embodiment.

FIG. 4 c is an illustration of yet another example display, according toan example embodiment.

FIG. 5 is an illustration of an example intensity level meter, accordingto an example embodiment.

FIG. 6 illustrates an example system for receiving, transmitting, anddisplaying data.

FIG. 7 illustrates an alternate view of the system illustrated in FIG.6.

FIG. 8 a illustrates an example system for receiving, transmitting, anddisplaying data.

FIG. 8 b illustrates an example system for receiving, transmitting, anddisplaying data.

FIG. 9 illustrates a schematic drawing of an example computer networkinfrastructure.

FIG. 10 a is an illustration of an example sound-event indicator,according to an example embodiment.

FIG. 10 b is an illustration of another example sound-event indicator,according to an example embodiment.

DETAILED DESCRIPTION

The following detailed description describes various features andfunctions of the disclosed systems and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context dictates otherwise. The illustrativesystem and method embodiments described herein are not meant to belimiting. It will be readily understood that certain aspects of thedisclosed systems and methods can be arranged and combined in a widevariety of different configurations, all of which are contemplatedherein.

I. Overview

A wearable computing system may be configured to allow visual perceptionof a real-world environment and to display computer-generatedinformation related to the perception of the real-world environment.Advantageously, the computer-generated information may be integratedwith a user's perception of the real-world environment. For example, thecomputer-generated information may supplement a user's perception of thephysical world with useful computer-generated information or viewsrelated to what the user is perceiving or experiencing at a givenmoment.

In some situations, a user of a wearable computing system may havedifficulty hearing. For instance, the user may be hearing impaired, suchas hard of hearing or even deaf. As such, it may be difficult for a userto hear the sound of the surrounding environment. Thus, it may bebeneficial to provide an indication of the sound of the surroundingenvironment to the user. For instance, it may be beneficial to providean indication of a direction from the wearable computing system of asource of sound and/or an intensity of the sound. As an example, a usermay be at a crosswalk attempting to cross a street, and an oncoming carmay be honking at the user in order to alert to the user that the car isdriving through the crosswalk. In such a case, it may be helpful toindicate to the user the direction from which the honk is coming (e.g.,from the left or the right), and the intensity of the honk (e.g., inorder to indicate how close the oncoming car it to the user).

The methods and systems described herein can facilitate providing anindication of sound present in the surrounding real-world environment.An example method may involve: (a) receiving audio data corresponding tosound detected by a wearable computing system; (b) analyzing the audiodata to determine both (i) a direction from the wearable computingsystem of a source of the sound and (ii) an intensity level of thesound; and (c) causing the wearable computing system to display one ormore indications that indicate (i) the direction of the source of thesound and (ii) the intensity level of the sound. By displaying these oneor more indications, a user may beneficially see indications of thesounds of the surrounding environment.

In accordance with an example embodiment, the wearable computing systemmay condition displaying the one or more indications on the intensitylevel being above a given threshold level. Further, many types of soundindications are possible. As an example, the indication may be a singlegraphic indication (e.g., an arrow) that indicates both (i) thedirection of the source from the wearable computing system and (ii) theintensity level of the sound. As another example, the wearable computingsystem may display a first indication that indicates the direction ofthe source of the sound (e.g., an arrow) and (ii) a second indicationthat indicates the intensity level of the sound (e.g., an intensitymeter).

II. Exemplary Methods

Exemplary methods may involve a wearable computing system displaying oneor more sound indications for sound of the surrounding environment. FIG.1 is a flow chart illustrating a method according to an exampleembodiment. More specifically, example method 100 involves receivingaudio data corresponding to sound detected by a wearable computingsystem, as shown by block 102. The method may then involve analyzing theaudio data to determine both (i) a direction from the wearable computingsystem of a source of the sound and (ii) an intensity level of thesound, as shown by block 104. Further, the method may then involvecausing the wearable computing system to display one or more indicationsthat indicate (i) the direction of the source of the sound and (ii) theintensity level of the sound, as shown by block 106.

Although the exemplary method 100 is described by way of example asbeing carried out by a wearable computing system, such as wearablecomputing system 600, 800, or 820, it should be understood that anexample method may be carried out by a wearable computing device incombination with one or more other entities, such as a remote server incommunication with the wearable computing system.

A. Receiving Audio Data Corresponding to Sound Detected by a WearableComputing System

As mentioned above, at block 102 the wearable computing system mayreceive audio data corresponding to the detected sound. In an example,the wearable computing system may include a plurality of microphonesconfigured to detect the sound of the surrounding environment. Note thatas used throughout the specification, “sound” may include a singularsound event present in the surrounding environment (e.g., a singlethunder strike, a single musical note) or a plurality of sounds presentin the surrounding environment (e.g., multiple thunder strikes, a song).In addition, the sound of the surrounding environment may come from asingle sound source or multiple sound sources.

The plurality of microphones configured to detect the sound of thesurrounding environment may be arranged in a manner to detect soundcoming from a plurality of directions. For instance, the microphones maybe arranged in a microphone array. FIG. 2 illustrates an examplewearable computing system 200 having a microphone array 202. Inparticular, computing system 200 includes an array 202 of directionalmicrophones 204 a through 204 e. Each directional microphone 204 athrough 204 e is arranged so as to capture audio in its respectivecorresponding region 206 a through 206 e. Note that although eachdirectional microphone is positioned to primarily detect sound comingfrom the corresponding region, the microphones may also detect soundcoming from other regions.

In an example, the microphones of array 202 may be arranged alongvarious points of the frame or frames of the wearable computing system,such as frames 604, 606, and 608 shown in FIG. 6. Further, note thatwhile the array 202 is arranged to detect sounds coming from regionsthat span approximately 150 degrees, an array in accordance with anotherexample embodiment may detect sounds coming from regions that span moredegrees (e.g., up to 360 degrees) or fewer degrees.

In an example, with reference to FIG. 2, the surrounding environment 207of wearable computing system 200 may include multiple sources of sound.For instance, a first source 208 of sound 210 may be located in region206 a, and a second source 212 of sound 214 may be located in region 206d. The wearable computing system 200 may receive audio datacorresponding to the sound of the surrounding environment. Inparticular, the microphone array 202 may detect sound 210 and sound 214.In an example, each microphone may detect the sound 210 and 214. Due tothe arrangement of the microphones in the array 202, each directionalmicrophone 204 a-204 e may detect the sound 210 and 214 at differenttimes and/or at different intensity levels.

B. Analyzing the Audio Data

After detecting the sound of the surrounding environment, the wearablecomputing system may send the audio data to a processor, such asprocessor 914 shown in FIG. 9, for analysis. In particular, the wearablecomputing system may, at block 104, analyze the audio data to determineboth (i) a direction from the wearable computing system of a source ofthe sound and (ii) an intensity level of the sound.

In order to determine the direction from the wearable computing systemof the source of the sound, the wearable computing system may analyzethe audio data collected by the microphones 204 a-204 e. As mentionedabove, in an example, each microphone may detect sound 210 and sound214. However, each directional microphone 204 a-204 e may detect a sound210 or 214 at a different time and/or at a different intensity level.For instance, sound 210 will likely be detected by microphone 204 aprior to sound 210 being detected by microphones 204 b-e. Similarly,sound 214 will likely be detected by microphone 204 d prior to sound 214being detected by 204 a-c and 204 e. Based on the time differencebetween the detection of the sound at the various microphones, thewearable computing system may determine the direction of the source ofthe sound. Such directional analysis of audio data is well known in theart. Based on the audio data detected by array 202, the system maydetermine that the source 208 of sound 210 is located in region 206 a,and that the source 212 of sound 214 is located in region 206 d.

Further, the wearable computing system may analyze the audio data todetermine the intensity level (e.g., volume) of the sound. Determiningthe intensity level of sound is well known in the art. In an example,the intensity level may be the intensity level of the sound detected bythe microphone of the region in which the sound is located. In anotherexample, the intensity level may be an average intensity level of thesound as received by each microphone in the array.

Note that the intensity level of the sound of the surroundingenvironment may vary with time. For instance, source 208 may be a persontalking, and the volume of the user's voice may change over time. Asanother example, the source may be a musical instrument outputting soundthat varies in intensity over time. Other examples are possible as well.

Further, intensity measurements may aid with determining the directionof the source of the sound, as the microphones closer in proximity tothe source will likely record the sound at a slightly higher intensitylevel.

C. Causing the Wearable Computing System to Display One or MoreIndications that Indicate (i) the Direction of the Source of the Soundand (ii) the Intensity Level of the Sound

Responsive to analyzing the audio data, the wearable computing systemmay, at block 106, display one or more indications that indicate thedirection of the source of the sound and the intensity level of thesound. The one or more indications beneficially may provide to the uservaluable information related to the sound of the surroundingenvironment. For instance, these displayed indications may help to alerta hard-of-hearing user of the sound present in the surroundingenvironment. In another example, these displayed indications may help ahard-of-hearing user visualize the surrounding environment as it sounds.

i. Single Graphic Indication

The one or more indications may comprise a single graphic indicationthat indicates both (i) the direction of the source from the wearablecomputing system and (ii) the intensity level. For instance, the singlegraphic indication may be an arrow.

FIG. 3 a illustrates an example display 300 of wearable computing system200. In this example, source 208 is the only source of sound at the timeof display 300 (i.e., source 212 is not making any sound at this time).Display 300 includes a single graphic indication for sound 210. Inparticular, in FIG. 3 a, the source 208 of sound 210 is a personspeaking. The single graphic indication of arrow 302 indicates both (i)the direction of the source from the wearable computing system and (ii)the intensity level. In this example, arrow 302 is located near thesource of the sound. In particular, the arrow 302 is located slightlyabove the source of the sound. The arrow may serve as a clear indicationto the user of the direction from which the sound is coming. In otherexamples, the arrow could be located slightly to the left or right ofthe source, or perhaps overlaid over the source 208.

As mentioned above, the intensity level of the sound may vary with time.Thus, in an example, a size of the arrow varies based on the varyingintensity level. For instance, a human voice may range from 0 decibelsto 60 decibels (dB), depending on how loud the person is speaking. Thesize of the arrow in the display may change based on the intensity levelof the sound 210. FIG. 3 b depicts example arrow sizes that may beincluded in display 300. In this example, arrow 310 is correlated with60 dB, arrow 312 is correlated with 50 dB, arrow 314 is correlated with40 dB, arrow 316 is correlated with 30 dB, arrow 318 is correlated with20 dB, and arrow 320 is correlated with 10 dB. Of course, arrows forother decibel levels between those mentioned may be included as well. Bydisplaying a larger arrow for larger intensity levels, the user of thewearable computing system will know when a loud sound is present in thesurrounding environment. In an example, the louder a sound in thesurrounding environment, the more likely it is that a user should payattention to or be aware of the sound. Thus, when a user sees a largearrow displayed, this display will indicate to the user to payparticular attention to the surrounding environment.

As mentioned above, there may be two or more sources of sound in thesurround environment. The wearable computing system may thus receiveaudio data corresponding to each of the plurality of sounds, and mayanalyze the audio data to determine both (i) a direction from thewearable computing system of each source of the sound and (ii) anintensity level of each sound. The wearable computing system may thendisplay one or more indications for each sound to indicate the directionof the sound and the intensity level of the sound. For instance, asshown in FIG. 4 a, person 208 and person 212 may be talking at the sametime. FIG. 4 a depicts an example display 400 that includes a firstindication 402 for sound 210, and a second indication 404 for sound 214.In this example, the indications are halos above the respectivespeakers.

These halos indicate both the direction of the source of the sound, andmay also indicate the intensity. In an example, the size of the halocould change for different intensities. However, in another example,different colors may be displayed for different intensity levels of thesound. For example, the halos could be color-coded to indicate varyingintensity levels. For example, FIG. 4 b depicts a scale 408 of differentcolor codes. In this example, color 410 is correlated with 60 dB, color412 is correlated with 50 dB, color 414 is correlated with 40 dB, color416 is correlated with 30 dB, color 418 is correlated with 20 dB, andcolor 420 is correlated with 10 dB. In an example, brighter colors maybe correlated with higher intensity levels.

As another example, the indication for intensity may include acolor-coded sound map. For example, FIG. 4 c illustrates an exampledisplay 450 with a color-coded sound map. In particular, the areas ofthe display that correspond to the source of sound may be colored toindicate the intensity level of the sound. Further, areas of the displaythat do not correspond to sources of the sound may not be colored (i.e.,they may remain their natural color as perceived by the user). In thisexample, the source 212 is color 412, while the source 208 is color 416.Thus, the user will be aware that the sound from source 212 is louderthan the sound from source 208 (i.e., the person 212 is talking moreloudly than person 208).

As yet another example, an example single graphic indication may be aflashing light disposed on or near the source of sound. The location ofthe flashing light may indicate the direction of the source from thewearable computer, and the frequency of the flashing light may beadjusted to indicate various intensity levels.

As another example, the indication for intensity may include a circularindicator. For instance, a graphical indicator could be placed at alocation on the circular indicator corresponding to the arrival angle ofthe sound. Further, a distance from the center of the circle couldcorrespond to the sound intensity level. A circular indicator may beuseful, for example, in the instance of a system capable of a capturingarrival angle through all 360 degrees. In an example, the circularindicator may be displayed in a periphery of a display screen of awearable computing system, such as in one of the corners of the displayscreen.

FIGS. 10 a and 10 b illustrate example circular indicators. Inparticular, FIG. 10 a illustrates an example circular indicator 1000.The circular indicator 1000 includes a circle 1002 and an intensity bar1004. In this example, the location of the intensity bar 1004 may serveto indicate that the source of the sound is directly behind the user.Further, the length of the intensity bar 1004 from the center 1006 ofthe circle 1002 may serve to indicate the intensity level of the sound.For instance, the intensity bar may be shorter (i.e., further from thecenter) for a less intense sound, and the intensity bar may be longer(i.e., closer to the center) for a more intense sound. FIG. 10 billustrates another example circular indicator 1010. In this example,the circular indicator includes a circle 1012 and an intensity bar 1014.The intensity bar 1014 serves to indicate that the source of the soundis to the left of the user. Further, the length of the intensity bar1014 indicates the intensity level of the sound. Compared to the soundintensity displayed in FIG. 10 a, the intensity of the sound displayedin FIG. 10 b is lower.

In an example, rather than including an intensity bar, the circularindicator may include an arrow that may be sized to indicate theintensity of the sound. In yet another example, the circular indicatormay include a graphic located on the circle (e.g., a dot), and the dotmay change color based on the intensity of the sound. It should beunderstood that the circular indicator may include other graphics thatcould serve to indicate the intensity level of the sound.

ii. Separate Indications for Direction and Intensity

In another example, the one or more indications may comprise a firstindication that indicates the direction of the source of the sound and asecond indication that indicates the intensity level of the sound. Forinstance, the first indication may be an arrow or a halo indicating thedirection of the source of the sound, and the second indication may bean intensity meter or a color-coded sound map. An example intensitymeter 500 is depicted in FIG. 5. In the example, the intensity meterranges from 0 db to 150 decibels, and a bar 502 indicates the intensitylevel of the sound (which, in this example, is 100 dB). The intensitylevel of the sound may vary with time, and thus the size of bar 502 mayvary in phase with the intensity level.

An intensity meter such as intensity meter 502 could be display atvarious locations in the display. For example, the intensity meter couldbe displayed in the periphery of the display. In another example, theintensity meter could be displayed near the source of the sound.Further, in the case where there are multiple sources of sound thewearable computing system may display an intensity meter for each sourceof sound.

iii. Sources of Sound Outside the User's Field of View

In an example, the source of the sound may be outside a field of view ofthe user. The wearable computing system may provide a view of areal-world environment of the wearable computing system that spansapproximately 180 degrees. The wearable computing system may detect asound coming from a source that is outside of this view of thesurrounding environment. For example, the source of the sound may bebehind the user, or to the left or right of the user out of the user'speripheral vision. The wearable computing system may determine that thesource of the sound is located either outside the view or in a peripheryof the view. In an example, the wearable computing system may include aplurality of microphones placed around the wearable computing system,such that the microphones can detect sounds from any direction (i.e.,spanning 360 degrees). For instance, the wearable computing system mayinclude a triangular placement of three microphones spaced 120 degreesaround the head of the user. Other microphone placements are possible aswell.

An example indication may be a flashing light flashing light in theperiphery of the view. For example, the display may include a flashinglight at some point in the periphery of the display, such as at point470 in the periphery 472 of display 450. This flashing light mayindicate to the user that a sound was detected outside of the user'sview.

iv. Displaying Sounds above a Threshold Intensity Level

In an example, the method may involve conditioning causing the wearablecomputing system to display the one or more indications on the intensitylevel being above a given threshold level. Thus, after determining theintensity level of the sound of the surrounding environment, thewearable computing system may determine whether the intensity levelexceeds a given threshold level. If the intensity level exceed the giventhreshold level, the wearable computing system may display the one ormore indications. However, if the intensity level does not exceed thegiven threshold level, the wearable computing system may not display theone or more indications.

This conditioning step may be useful, for instance, in filtering outbackground noise, of which the user may not be interested in seeing anindication. In an example, the wearable computing system conditionscausing the wearable computing system to display the one or moreindications on the intensity level being above 5 dB. Other examplethresholds are possible, and the user may configure the wearablecomputing system with a desired threshold level, such as a thresholdlevel between 1 dB and 20 dB.

In another example, a user may be wearing the wearable computing system,and a display of the wearable computing system may be initially poweredoff. For example, the display may be off for one reason or another.However, while the display is powered off, the wearable computing systemmay detect a sound. The system may determine that the intensity level isabove a given threshold level. Responsive to determining that theintensity level is above the given threshold level, the system mayactivate the display in order to display the one or more indications.

v. Displaying Intensity Level Relative to Moving Average Window

In an example, rather than indicating the absolute intensity of a sound,the display of the intensity level of a sound event may be representedas the difference between the absolute intensity level of the soundevent and that of a moving average window of the sound of thesurrounding environment. The moving average window may be a movingaverage window of a predetermined duration of time. For instance, themoving average window may be 30 seconds. Of course, the moving averagewindow could be of a longer or shorter duration.

By representing the intensity level of a sound event as the differencebetween the absolute intensity level of the sound event and that of amoving average window of the sound of the surrounding environment, thesound intensity displayed may be tailored to the particular environmentin which a user is located. This may help serve to indicate to the userrelevant sounds in the user's particular sound environment. For example,if a user of a wearable computing system is in a very quiet environment(e.g., an empty room), the last 30 seconds of average sound power willlikely be low. An aberrant event could be something as quiet as a lightknock on the door. The sound intensity of the light knock may bedisplayed relative to the likely low sound power of the last 30 secondsof average sound power.

On the other hand, if a user of a wearable computing system is in a verynoisy environment (e.g., a loud factory), the last 30 seconds of averagesound power will be high. Sound intensity levels may be displayedrelative to the average sound intensity of the very noisy environment.In addition, as mentioned above, in some examples, the wearablecomputing system may condition displaying a sound intensity being abovea given threshold. The given threshold may be based on an average noisefloor. For instance, the given threshold for conditioning displaying asound intensity may be greater for a loud environment than for a quietenvironment. Using the examples discussed above, in the quietenvironment, the average noise floor may be set at a level near theaverage noise level of the quiet room. On the other hand, in the loudenvironment, the average noise floor may be set at a level near theaverage noise of the loud factory. Beneficially, by adjusting thethreshold level based on an average noise floor, the wearable computingsystem may ensure that it displays sounds that are relevant to theparticular sound environment in which a user is located.

In an example, the wearable computing system may be configured tocontinuously run this moving-average analysis. By running themoving-average analysis continuously, a user would not have to manuallychange the threshold level when the user moves to a different soundenvironment (e.g., from a quiet environment to a loud environment).

vi. Additional Indications of Sound

In addition to displaying indications of the direction of the source ofthe sound from the wearable computing system and the intensity level ofthe sound, other sound indications are also possible. As mentionedabove, the sound of the surrounding environment may be speech. In anexample, the method may further comprise a speech-to-text featuredetermining text of the speech. After determining the text of thespeech, the method may involve causing the wearable computing system todisplay the text of the speech. In an example, the wearable computingsystem may display a speech bubble above the person speaking, where thespeech bubble includes the text of the speaker's speech.

Another possible sound indication is an indication that indicates whatthe source of the sound is. In an example, the wearable computing systemmay determine the source of the sound, and the wearable computing systemmay provide information regarding the source of the sound. The wearablecomputing system may analyze the audio data to determine the source ofthe sound, and may then display another indication that indicates thesource of the sound. For example, the other indication may be a textindication that describes the source of the sound (e.g., a dog, a cat, ahuman, a musical instrument, a car, etc.).

For example, the wearable computing system may analyze the sound tomatch it to a source of the sound. In an example, a remote server mayinclude a database of sound snippets, each snippet being correlated witha source. The server may compare the detected sound to various sounds inthe database of sound snippets to identify a match. Once identifying thesource of the sound, the wearable computing system may displayinformation about the source. For example, may display informationidentifying the source. For example, the sound may sound from an engineof a given vehicle. The wearable computing system may display the typeof vehicle it is. Other information may be provided. For example, thewearable computing system may display information related to the marketprice of the vehicle and/or establishments where the vehicle can bepurchased. Other example information is possible as well.

In another example, the wearable computing system may display a videothat corresponds to the source of the sound. For example, the wearablecomputing system may detect sounds from a beach (e.g., waves crashing).The wearable computing system could display a video of a beach and wavescrashing. This video may help the user visualize the world as it sounds.

In yet another example, another possible sound indication is anindication that indicates the audio frequency of the sound. Audiofrequency is a property of sound that, for example, determines pitch,and the audio frequency may be measured in hertz (Hz). Generally, humanstypically are capable of hearing frequencies in the range of 20 Hz to20,000 Hz. However, the range of frequencies individuals hear may beinfluenced by environmental factors.

The wearable computing system may analyze the sound and determine theaudio frequency of the sound in a variety of ways. For example, thewearable computing system may analyze the audio frequency of the soundby algorithms either in frequency domain or time domain. The former mayattempt to locate the sinusoidal peaks in the frequency transform of theinput signal. The latter may use autocorrelation functions to detectsimilarities between the waveform and the time-lagged version of it.After determining the audio frequency of the sound, the wearablecomputing system may display an indication of the determined audiofrequency. For example, the wearable computing system may display thedetermined audio frequency value. As another example, the wearablecomputing system may display an audio frequency scale.

Determining the audio frequency of a sound and displaying an indicationof the audio frequency may be beneficial for a variety of reasons. Anexample benefit of being able to visualize audio frequency may beexperienced in a condition where a user is exposed to high-frequencynoises. In such a condition, the display of an indication of a highaudio frequency can help warn the user of a sound having a high audiofrequency. This warning may, for example, may serve to influence theuser to take action to prevent potential hearing impairment due to thehigh audio frequency sound.

In still yet another example, another possible sound indication is anindication that indicates Wiener entropy. Wiener entropy can be analyzedas the ratio of geometric mean to arithmetic mean of the sound spectrum.In particular, Wiener entropy is a measure of randomness of sound on ascale between 0 and 1, which can be used to differentiate whether thesound is a white noise or a harmonic sound. Thus, a user (e.g., a userthat is hard of hearing) of the wearable computing system maybeneficially determine whether a sound is a white noise or a harmonicsound.

In addition to the example sound indications described above, otherindications of the sound are possible as well.

III. Example Systems and Devices

FIG. 6 illustrates an example system 600 for receiving, transmitting,and displaying data. The system 600 is shown in the form of a wearablecomputing device. System 600 may be configured to carry out method 100.While FIG. 6 illustrates a head-mounted device 602 as an example of awearable computing device, other types of wearable computing devicescould additionally or alternatively be used. As illustrated in FIG. 6,the head-mounted device 602 comprises frame elements includinglens-frames 604, 606 and a center frame support 608, lens elements 610,612, and extending side-arms 614, 616. The center frame support 608 andthe extending side-arms 614, 616 are configured to secure thehead-mounted device 602 to a user's face via a user's nose and ears,respectively.

Each of the frame elements 604, 606, and 608 and the extending side-arms614, 616 may be formed of a solid structure of plastic and/or metal, ormay be formed of a hollow structure of similar material so as to allowwiring and component interconnects to be internally routed through thehead-mounted device 602. Other materials may be possible as well.

One or more of each of the lens elements 610, 612 may be formed of anymaterial that can suitably display a projected image or graphic. Each ofthe lens elements 610, 612 may also be sufficiently transparent to allowa user to see through the lens element. Combining these two features ofthe lens elements may facilitate an augmented reality or heads-updisplay where the projected image or graphic is superimposed over areal-world view as perceived by the user through the lens elements.

The extending side-arms 614, 616 may each be projections that extendaway from the lens-frames 604, 606, respectively, and may be positionedbehind a user's ears to secure the head-mounted device 602 to the user.The extending side-arms 614, 616 may further secure the head-mounteddevice 602 to the user by extending around a rear portion of the user'shead. Additionally or alternatively, for example, the system 600 mayconnect to or be affixed within a head-mounted helmet structure. Otherpossibilities exist as well.

The system 600 may also include an on-board computing system 618, avideo camera 620, a sensor 622, and a finger-operable touch pad 624. Theon-board computing system 618 is shown to be positioned on the extendingside-arm 614 of the head-mounted device 602; however, the on-boardcomputing system 618 may be provided on other parts of the head-mounteddevice 602 or may be positioned remote from the head-mounted device 602(e.g., the on-board computing system 618 could be wire- orwirelessly-connected to the head-mounted device 602). The on-boardcomputing system 618 may include a processor and memory, for example.The on-board computing system 618 may be configured to receive andanalyze data from the video camera 620 and the finger-operable touch pad624 (and possibly from other sensory devices, user interfaces, or both)and generate images for output by the lens elements 610 and 612.

The video camera 620 is shown positioned on the extending side-arm 614of the head-mounted device 602; however, the video camera 620 may beprovided on other parts of the head-mounted device 602. The video camera620 may be configured to capture images at various resolutions or atdifferent frame rates. Many video cameras with a small form-factor, suchas those used in cell phones or webcams, for example, may beincorporated into an example of the system 600.

Further, although FIG. 6 illustrates one video camera 620, more videocameras may be used, and each may be configured to capture the sameview, or to capture different views. For example, the video camera 620may be forward facing to capture at least a portion of the real-worldview perceived by the user. This forward facing image captured by thevideo camera 620 may then be used to generate an augmented reality wherecomputer generated images appear to interact with the real-world viewperceived by the user.

The sensor 622 is shown on the extending side-arm 616 of thehead-mounted device 602; however, the sensor 622 may be positioned onother parts of the head-mounted device 602. The sensor 622 may includeone or more of a gyroscope or an accelerometer, for example. Othersensing devices may be included within, or in addition to, the sensor622 or other sensing functions may be performed by the sensor 622.

The finger-operable touch pad 624 is shown on the extending side-arm 614of the head-mounted device 602. However, the finger-operable touch pad624 may be positioned on other parts of the head-mounted device 602.Also, more than one finger-operable touch pad may be present on thehead-mounted device 602. The finger-operable touch pad 624 may be usedby a user to input commands. The finger-operable touch pad 624 may senseat least one of a position and a movement of a finger via capacitivesensing, resistance sensing, or a surface acoustic wave process, amongother possibilities. The finger-operable touch pad 624 may be capable ofsensing finger movement in a direction parallel or planar to the padsurface, in a direction normal to the pad surface, or both, and may alsobe capable of sensing a level of pressure applied to the pad surface.The finger-operable touch pad 624 may be formed of one or moretranslucent or transparent insulating layers and one or more translucentor transparent conducting layers. Edges of the finger-operable touch pad624 may be formed to have a raised, indented, or roughened surface, soas to provide tactile feedback to a user when the user's finger reachesthe edge, or other area, of the finger-operable touch pad 624. If morethan one finger-operable touch pad is present, each finger-operabletouch pad may be operated independently, and may provide a differentfunction.

FIG. 7 illustrates an alternate view of the system 600 illustrated inFIG. 6. As shown in FIG. 7, the lens elements 610, 612 may act asdisplay elements. The head-mounted device 602 may include a firstprojector 628 coupled to an inside surface of the extending side-arm 616and configured to project a display 630 onto an inside surface of thelens element 612. Additionally or alternatively, a second projector 632may be coupled to an inside surface of the extending side-arm 614 andconfigured to project a display 634 onto an inside surface of the lenselement 610.

The lens elements 610, 612 may act as a combiner in a light projectionsystem and may include a coating that reflects the light projected ontothem from the projectors 628, 632. In some embodiments, a reflectivecoating may not be used (e.g., when the projectors 628, 632 are scanninglaser devices).

In alternative embodiments, other types of display elements may also beused. For example, the lens elements 610, 612 themselves may include: atransparent or semi-transparent matrix display, such as anelectroluminescent display or a liquid crystal display, one or morewaveguides for delivering an image to the user's eyes, or other opticalelements capable of delivering an in focus near-to-eye image to theuser. A corresponding display driver may be disposed within the frameelements 604, 606 for driving such a matrix display. Alternatively oradditionally, a laser or LED source and scanning system could be used todraw a raster display directly onto the retina of one or more of theuser's eyes. Other possibilities exist as well.

FIG. 8 a illustrates an example system 800 for receiving, transmitting,and displaying data. System 800 may be configured to carry out method100. The system 800 is shown in the form of a wearable computing device802. The wearable computing device 802 may include frame elements andside-arms such as those described with respect to FIGS. 6 and 7. Thewearable computing device 802 may additionally include an on-boardcomputing system 804 and a video camera 806, such as those describedwith respect to FIGS. 6 and 7. The video camera 806 is shown mounted ona frame of the wearable computing device 802; however, the video camera806 may be mounted at other positions as well.

As shown in FIG. 8 a, the wearable computing device 802 may include asingle display 808 which may be coupled to the device. The display 808may be formed on one of the lens elements of the wearable computingdevice 802, such as a lens element described with respect to FIGS. 6 and7, and may be configured to overlay computer-generated graphics in theuser's view of the physical world. The display 808 is shown to beprovided in a center of a lens of the wearable computing device 802,however, the display 808 may be provided in other positions. The display808 is controllable via the computing system 804 that is coupled to thedisplay 808 via an optical waveguide 810.

FIG. 8 b illustrates an example system 820 for receiving, transmitting,and displaying data. System 820 may be configured to carry out method100. The system 820 is shown in the form of a wearable computing device822. The wearable computing device 822 may include side-arms 823, acenter frame support 824, and a bridge portion with nosepiece 825. Inthe example shown in FIG. 8 b, the center frame support 824 connects theside-arms 823. The wearable computing device 822 does not includelens-frames containing lens elements. The wearable computing device 822may additionally include an on-board computing system 826 and a videocamera 828, such as those described with respect to FIGS. 6 and 7.

The wearable computing device 822 may include a single lens element 830that may be coupled to one of the side-arms 823 or the center framesupport 824. The lens element 830 may include a display such as thedisplay described with reference to FIGS. 6 and 7, and may be configuredto overlay computer-generated graphics upon the user's view of thephysical world. In one example, the single lens element 830 may becoupled to the inner side (i.e., the side exposed to a portion of auser's head when worn by the user) of the extending side-arm 823. Thesingle lens element 830 may be positioned in front of or proximate to auser's eye when the wearable computing device 822 is worn by a user. Forexample, the single lens element 830 may be positioned below the centerframe support 824, as shown in FIG. 8 b.

FIG. 9 illustrates a schematic drawing of an example computer networkinfrastructure. In system 900, a device 910 communicates using acommunication link 920 (e.g., a wired or wireless connection) to aremote device 930. The device 910 may be any type of device that canreceive data and display information corresponding to or associated withthe data. For example, the device 910 may be a heads-up display system,such as the head-mounted device 602, 800, or 820 described withreference to FIGS. 6-8 b.

Thus, the device 910 may include a display system 912 comprising aprocessor 914 and a display 916. The display 910 may be, for example, anoptical see-through display, an optical see-around display, or a videosee-through display. The processor 914 may receive data from the remotedevice 930, and configure the data for display on the display 916. Theprocessor 914 may be any type of processor, such as a micro-processor ora digital signal processor, for example.

The device 910 may further include on-board data storage, such as memory918 coupled to the processor 914. The memory 918 may store software thatcan be accessed and executed by the processor 914, for example.

The remote device 930 may be any type of computing device or transmitterincluding a laptop computer, a mobile telephone, or tablet computingdevice, etc., that is configured to transmit data to the device 910. Theremote device 930 and the device 910 may contain hardware to enable thecommunication link 920, such as processors, transmitters, receivers,antennas, etc.

In FIG. 9, the communication link 920 is illustrated as a wirelessconnection; however, wired connections may also be used. For example,the communication link 920 may be a wired serial bus such as a universalserial bus or a parallel bus. A wired connection may be a proprietaryconnection as well. The communication link 920 may also be a wirelessconnection using, e.g., Bluetooth® radio technology, communicationprotocols described in IEEE 802.11 (including any IEEE 802.11revisions), Cellular technology (such as GSM, CDMA, UMTS, EV-DO, WiMAX,or LTE), or Zigbee® technology, among other possibilities. The remotedevice 330 may be accessible via the Internet and may include acomputing cluster associated with a particular web service (e.g.,social-networking, photo sharing, address book, etc.).

With reference to FIG. 9, device 910 may perform the steps of method100. In particular, method 100 may correspond to operations performed byprocessor 914 when executing instructions stored in a non-transitorycomputer readable medium. In an example, the non-transitory computerreadable medium could be part of memory 918. The non-transitory computerreadable medium may have instructions stored thereon that, in responseto execution by processor 914, cause the processor 914 to performvarious operations. The instructions may include: (a) instructions forreceiving audio data corresponding to sound detected by a wearablecomputing system; (b) instructions for analyzing the audio data todetermine both (i) a direction from the wearable computing system of asource of the sound and (ii) an intensity level of the sound; and (c)instructions for causing the wearable computing system to display one ormore indications that indicate (i) the direction of the source of thesound and (ii) the intensity level of the sound. Further, in an example,the instructions may further include instructions for conditioningcausing the wearable computing system to display the one or moreindications on the intensity level being above a given threshold level.

IV. Conclusion

It should be understood that arrangements described herein are forpurposes of example only. As such, those skilled in the art willappreciate that other arrangements and other elements (e.g. machines,interfaces, functions, orders, and groupings of functions, etc.) can beused instead, and some elements may be omitted altogether according tothe desired results. Further, many of the elements that are describedare functional entities that may be implemented as discrete ordistributed components or in conjunction with other components, in anysuitable combination and location.

It should be understood that for situations in which the systems andmethods discussed herein collect and/or use any personal informationabout users or information that might relate to personal information ofusers, the users may be provided with an opportunity to opt in/out ofprograms or features that involve such personal information (e.g.,information about a user's preferences). In addition, certain data maybe anonymized in one or more ways before it is stored or used, so thatpersonally identifiable information is removed. For example, a user'sidentity may be anonymized so that the no personally identifiableinformation can be determined for the user and so that any identifieduser preferences or user interactions are generalized (for example,generalized based on user demographics) rather than associated with aparticular user.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims, along with the fullscope of equivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

What is claimed is:
 1. A computer-implemented method comprising:receiving audio data corresponding to sound detected by a wearablecomputing system; analyzing the audio data to determine (i) a directionfrom the wearable computing system of a source of the sound and (ii) anintensity level of the sound; and causing the wearable computing systemto display one or more indications that indicate (i) the direction ofthe source of the sound and (ii) the intensity level of the sound,wherein the intensity level of the sound varies with time, and whereinthe one or more indications comprise an indication associated with theintensity level that varies in phase with the intensity level.
 2. Themethod of claim 1, wherein a display of the wearable computing system isinitially powered off, the method further comprising: after determiningthe intensity level, determining that the intensity level is above agiven threshold level; and responsive to determining that the intensitylevel is above the given threshold level, activating the display inorder to display the one or more indications.
 3. The method of claim 1,wherein the one or more indications comprise a single graphic indicationthat indicates both (i) the direction of the source from the wearablecomputing system and (ii) the intensity level.
 4. The method of claim 3,wherein the single graphic indication is an arrow.
 5. The method ofclaim 4, wherein a size of the arrow varies based on the varyingintensity level.
 6. The method of claim 1, wherein the one or moreindications comprise (i) a first indication that indicates the directionof the source of the sound and (ii) a second indication that indicatesthe intensity level of the sound.
 7. The method of claim 6, wherein thefirst indication is an arrow, and wherein the second indication is anintensity meter.
 8. The method of claim 1, wherein the one or moreindications comprise a circular indicator having a graphical indicator,wherein a location of the graphical indicator on the circular indicatorcorresponds to an arrival angle of the sound and a distance of thegraphical indicator from a center of the circular indicator correspondsto the intensity level.
 9. The method of claim 1, wherein displaying theone or more indications comprises displaying different colors fordifferent intensity levels of the sound.
 10. The method of claim 1,wherein the one or more indications comprise an intensity meter.
 11. Themethod of claim 1, wherein the wearable computing system provides a viewof a real-world environment of the wearable computing system, the methodfurther comprising: determining that the source of the sound is locatedeither outside the view or in a periphery of the view, and wherein theone or more indications comprise a flashing light in the periphery ofthe view.
 12. The method of claim 1, wherein the sound comprises speech,the method further comprising: a speech-to-text feature determining textof the speech; and causing the wearable computing system to display thetext of the speech.
 13. The method of claim 1, further comprising:receiving audio data corresponding to a second sound detected by awearable computing system; analyzing the audio data to determine (i) adirection from the wearable computing system of a source of the secondsound, wherein direction of the second sound is different than thedirection of the sound, and (ii) an intensity level of the second sound;and causing the wearable computing system to display one or moreindications that indicate (i) the direction of the source of the secondsound and (ii) the intensity level of the second sound.
 14. The methodof claim 1, further comprising: analyzing the audio data to determinethe source of the sound; and causing the wearable computing system todisplay another indication, wherein the other indication indicates thesource of the sound.
 15. The method of claim 14, wherein the otherindication is a text indication that describes the source of the sound.16. The method of claim 14, wherein the other indication is a video thatcorresponds to the source of the sound.
 17. A non-transitory computerreadable medium having instructions stored thereon that, in response toexecution by a processor, cause the processor to perform operations, theinstructions comprising: instructions for receiving audio datacorresponding to sound detected by a wearable computing system;instructions for analyzing the audio data to determine (i) a directionfrom the wearable computing system of a source of the sound and (ii) anintensity level of the sound; instructions for causing the wearablecomputing system to display one or more indications that indicate (i)the direction of the source of the sound and (ii) the intensity level ofthe sound, wherein the intensity level of the sound varies with time,and wherein the one or more indications comprise an indicationassociated with the intensity level that varies in phase with theintensity level.
 18. The non-transitory computer readable medium ofclaim 17, wherein the one or more indications comprise a single graphicindication that indicates both (i) the direction of the source from thewearable computing system and (ii) the intensity level.
 19. A wearablecomputing system comprising: a head-mounted display, wherein thehead-mounted display is configured to display computer-generatedinformation and allow visual perception of a real-world environment; anda controller, wherein the controller is configured to receive audio datacorresponding to sound detected by a wearable computing system and toanalyze the audio data to determine (i) a direction from the wearablecomputing system of a source of the sound and (ii) an intensity level ofthe sound; and a display system, wherein the display system isconfigured to display one or more indications that indicate (i) thedirection of the source of the sound and (ii) the intensity level of thesound, wherein the intensity level of the sound varies with time, andwherein the one or more indications comprise an indication associatedwith the intensity level that varies in phase with the intensity level.20. The wearable computing system of claim 19, wherein the one or moreindications comprise a circular indicator having a graphical indicator,wherein a location of the graphical indicator on the circular indicatorcorresponds to an arrival angle of the sound and a distance of thegraphical indicator from a center of the circular indicator correspondsto the intensity level.